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\  ABSTRACT 

fit 

-W  dtttniat  the  conditions  under  which  sn  error  washing  schene 
based  on  strict  redundancy  can  be  need  to  increase  our  confidence  in  the 
results  of  parallel  computations.  This  study  shows  that  the  issues  of 
speed  and  reliability  of  parallel  processors  are  interdependent  and  must 
be  considered  jointly  at  the  design  stage. 
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i.  nmoDuenoN 


la  this  paper,  we  eoatinae  the  investigation,  began  in  [MEY83], 
into  the  nee  of  striet  hardware  redundancy  for  increasing  confidence  in 
the  results  of  parallel  computations.  As  before,  suppose  a  given  prob- 
lea  can  be  solved  in  a  given  tine  using  a  cluster  of  f)  computing  ele¬ 
ments,  and  suppose  the  following  hypothesis  holds: 

Bvno thesis  1: 

(i)  the  input  to  the  cluster  is  oorreot, 

(ii)  each  computing  element  in  the  cluster  has  the  same  probability  p  of 
remaining  non-faulty  during  some  given  "mission  time", 

(iii)  the  computing  elements  fail  independently. 

If  Pc  is  the  probability  that  the  output  of  the  cluster  is  correct, 
then  Hypothesis  1  implies 

pc  i  p"  -  pc.,-  (1> 

Note  that  a  computing  element  may  be  faulty  without  affecting  the  clus¬ 
ter  output.  If  ?£  is  not  high  enough  for  our  purposes,  one  way  to  try 
to  increase  our  confidence  in  the  cluster  output  is  to  accept  it  only 
when  we  think  it  is  correct.  This  approach  is  analysed  in  [MET83]. 
Another  approach  involves  the  use  of  an  error  masking  scheme  to  directly 
increase  the  probability  of  getting  correct  results.  Such  a  scheme  may 
be  implemented  by  replicating  the  original  cluster  u-1  times,  sending 
the  original  input  to  all  the  clusters,  and  then  sending  all  the  cluster 
outputs  to  an  error  masker  which,  by  coshering  them,  chooses  as  its  out- 


put  til*  one  aost  likely  to  be  correct.  Ike  aeeker  Bakes  its  choice  by 
first  partitioning  the  set  of  cluster  outputs  into  blocks  suoh  that  all 
the  outputs  in  a  given  block  are  identical,  and  then  by  choosing  any  of 
the  outputs  in  the  blook  of  aaxiaal  cardinality.  If  aore  than  one  blook 
has  aaxiaal  cardinality,  the  aasker  chooses  an  output  at  randoa  froa 
aaong  these  blocks.  The  quantity  of  interest  to  us  is  Pgg(a.p).  the 
probability  that  the  error  aasker  output  is  correct. 

In  this  paper,  we  shall  not  assuae  that  the  error  aasker  is  always 
non-faulty;  instead,  we  shall  assuae  that  the  error  aasker  always  pro¬ 
duces  the  desired  output  whenever  a  strict  aajority  of  its  inputs  are 
identical.  In  other  words,  we  shall  adopt  the  following  hypothesis: 

Hypothesis  £:  Whenever  a  strict  aajority  of  the  cluster  outputs  are 
identical,  the  output  of  the  error  aasker  is  equal  to  one  of  those  clus¬ 
ter  outputs. 

This  type  of  error  aasking  scheae  goes  back  to  von  Neuaann's  work 
in  the  Fifties  [NED63]  on  constructing  logic  devices.  His  work  was 
extended  and  applied  to  relay  circuits  by  Hoore  and  Shannon  [M0056] . 

See  [HAE65]  for  further  work  along  these  lines,  and  [AVI78]  for  a  sur¬ 
vey.  Most  authors  restrict  attention  to  the  ease  a  «  3  (triple  srodulsr 
redundancy)  -  see,  for  exaaple,  [LY062]  snd  [B0U71].  In  the  context  of 
parallel  processing,  no  eoaplete  analysis  of  the  usefulness  of  error 
Making,  under  any  reasonable  set  of  assuaptions,  has  been  esrried  out. 

The  analysis  presented  in  the  next  seotion  is  based  on  three  prin¬ 
ciples:  (i)  one  should  distinguish  between  hardware  faults  in  s 
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ooaputing  network  sad  inoorreot  resalts  produced  by  the  network;  (ii) 
oae  should  as  suae  ss  little  ss  possible  sbout  the  fsult  aechsaisa;  (iii) 
one  should  use  oaly  those  quantities  thst  have  soae  chsaoe  of  be ins 
experimentally  aeasured.  These  eonsiderstions  rule  out.  in  particular , 
the  use  of  s  failure  probability  distribution.  They  also  lead  us  to  a 
worst  esse  analysis. 


2.  ANALYSIS  OF  THE  ERROR  MASKING  SCHEME 


la  this  toot ion,  i  worst  csss  analysis  of  tie  error  narking  scheme 
is  presented.  Given  a  2  1  clusters,  each  eontaining  0  computing  ele¬ 
ments,  let  Ej(a,0)  he  the  event  that  exaetly  j  elnsters  are  faulty.  A 
cluster  is  faulty  if  and  only  if  at  least  one  of  its  elenents  is  faulty, 
and  therefore  the  probability  that  a  oluster  is  faulty  is  1-p^.  As  a 
result, 

PtEjU,*))  -  j1(%lj)t(l-p*)JpP(<|-J),  j  -  0,1 . .  (2) 

Let  Bj(a,fl)  be  the  event  that  exactly  j  cluster  outputs  are 
incorrect.  It  is  clear  that  if  no  aore  than  k  clusters  are  faulty,  then 
at  aost  k  cluster  outputs  can  be  incorrect,  and  thus 

k  k 

XP(B  (u.0))  1  I  P(E  (a. fi)).  (3) 

j-0  J  j«0  3 


A  lower  bound  for  Pg|(s,p)  will  now  be  derived. 


1:  Under  Hypotheses  1  and  2,  2  Pq,  B(a,0),  where 


W-<»  •  «  <*«• 


and 


(o-2)/2 


P  (.  B)  -  X  - *1 _ (l-uP)JuP(a”j) 

PCM,a(a'P>  *  j!(u-j)!(1^  '  9 


a  even. 


(4) 


(5) 


Proof:  By  definition 


Ppyta.p)  -  J  P(uiker  output  correct  i  B. (a,0) )P(B. (a,p) ) , 
j-0  3  3 


end  it  is  clear  from  Hypothesis  2  that  for  j  <  a/2 


P(aasker  output  correot  I  Bj(a,p))  -  1. 


Thus,  for  every  k  <  a/2 


k  k 

P^U.p)  1  XP(B(a.p))2  ZP(E.(a,p» 

j-0  J  j-0  3 


and  the  result  of  the  lease  follows  froa  (2). 


The  behavior  of  the  lover  bound  P_.  (a,p)  as  a  function  of  a  is 

dll 

characterised  in  the  next  four  leaaias.  First,  define  q  and  w  via 


w  -  (l-q)/q 


Loans  £:  Suppose  that  Hypotheses  1  and  2  are  satisfied.  If  a  is  odd, 
then 

(a+l)/2(w+ir<a+2)  , 

POI,n(o+2'p)  _  PCM,a(o,P>  “  ( (a-l)/2) ! ( (a+l)/2) !  <1-w)  * 

Proof:  The  definition  of  w  iaplies  that  q  -  l/(w+l)  and  we  aay  rewrite 
(4)  as 


<a-l)/2 

z 

j-0 


si 


PCM,a<0»p) 


j 1 (a-j) ! 


w^(w+l)  ° 


X  TTT^ryr  .jen>2. 

j«0  J ‘ '°  J' * 


It  follows  that 


p  r«+2  A)  -  ,u|1>-(a+2)  .(a+j)/2(o+2)(a+l)  J 
PQI»«(a+2*^  "  (W+1)  °f  *  j ! ( o+2- j ) !  ^ 


Thus, 


pcm,.(o+2'p)  ■  pai,«<0'p)  "  <*+i)‘(a+2>«»  x  Tj*j» 

j  -o 


(o+3 )/2 


where 


v  .  (a+2)(a+l)  _  _L 
T0  (o+2)!  a!  * 


v  .  Lfff  23  laj-U  .  J,  _ _ 1 _ 

Y1  (o+U!  a!  (o-l) !  * 


.  U+2JU+1) _ 1 _ 2 _  _  _ 1 _ 

Tj  j 1 (a+2-j ) !  (j-2)!(a-j+2)!  ( j-1) ! (o-j+1) !  jl(o-j)!' 


for  j  -  2.  3.....  (a-l)/2. 


v  .  ■  ii’amri/—  .  -  -  - -  i  .  . . .  - 

t(«+1)/2  ((o+1)/2) I ((a+3)/2) !  ((a-3)/2) ! ((a+3)/2) !  ( (o-l)/2) ! ( (a+l)/2) ! 


T(o+3)/2  "  "  ( (o-l)/2) ! ( (a+1) /2) ! 


It  Is  not  difficult  to  wsrify  that 


0,  j  ■  0,  If  2» , , m I 


(o-l)/2 
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_ _ 1 - 

t(«+1)/2  ((o-1)/2) ! ((o+l)/2) !  * 

Substituting  into  (6)  yields  tbs  desired  result. 

Lena*  1:  Suppose  tbst  Hypotheses  1  end  2  ere  sstisfied.  If  o  is  even, 
then 


Wtt+2**>  -  Wtt-P) 


wa/2(w+l)~(°'>'2)al 
( (a-2) /2) ! ( (a+2) /2) ! 


((a+2)/o-w)  . 


Proof:  Rewrite  (5)  ss 


(a-2)/ 2 

X  TT^rrr  ^(w+i)  ° 


PCM,«<0’P)  "  j I ( o- j ) I 


(o-2)/2 


It  follows  thst 


a/2. 


P  a\  „  i  (g+2)  (a+1)  j 

ra.m(a*2->>  •  (wtl>  alj*)jT(i+2-j)l  * 


Thus, 


(a+2)/2  . 

P«  »<«+2»P>  "  pm  "  <*♦*>  a!  S  r,*J» 

CM,«  CN,n  j 


(7) 


where 


„  U+2Ka+l)  _  _L 
T0  (a+2)  l  a!  * 


ia+21La±ll _ 2_  _ _ 1 _ 

T1  (a+l)l  a!  (a-l)l' 


j  j 1 (o+2-j ) !  <  j — 2) ! (o-j+2) !  ( j-1) ! (o-j+1) !  j!(o-j)! 


for  j  «  2.  3....,  (a-2)/2, 

_  _  _  (a+2) (a+1)  _ l _  _ 2 _ 

to/2  (a/2) ! ( (a+4) 12) !  ( (a-4)/2) I ( (a+4)/2) !  "  ((a-2)/2) ! ((a+2)/2) I 

and 


Y(a+2)/2 


_ l _ 

( (a-2) /2) 1 ( (a+2) /2) ! * 


It  it  not  difficult  to  verify  that 


Tj  -  0.  j  -  0.  1,  2.....  (a-2)/2. 


and 


v  .  _ ((ata)./g) _ 

Ta/2  ( (a-2)/2) ! ( (a+2)/2) !  * 

Substituting  into  (7)  yields  the  desired  result. 

Le—a  ±:  Suppose  that  Hypotheses  1  and  2  are  satisfied.  If  a  is  odd, 

* 

then 


( (a-1) /2) ! ( (a+1) /2) !  * 


Proof:  Since  a  is  odd,  we  nay  rewrite  (4)  and  (5)  as 


(a-l)/2 

Ppu  M(a,p)  -  (w+1)  '“'*'a!  Z 

j-0 


-(a+D_,  ""  i'" - 1 -  j 

j ! ( a- j ) !  W  1,1 


and 


CN,n 


(a+1,0)  -  (w+1) 


-(a+1) 


(a-l)/2 


a! 


(q+D 


j I ( a+1- j ) f 


PCM.»(o+1'P)  “  POi.-(a'P)  =  (W+1) 


-<a+l>  (o+J)/2  j 

o!  £  y.wJ 


where 


v  =  _Lfl±iL  _  _L 
T0  (a+l)!  a!  * 


rj  j ! (a+l-j ) !  ( j-1) ! (a-j+1) !  j!(a-j)!' 


foe  J  K  1 1  2 $  3 1 » •  e i  ( ct  l)/2«  ftfld 


t(o+1)/2  “  "  ((o-l)/2) !((o+l)/2) !* 


It  is  not  difficult  to  verify  that 


y j  *  0,  j  ■  0,  li  2f • • • *  ( u-1 ) /2, 


and  thus  (8)  reduces  to  the  desired  result. 


£:  Suppose  that  Hypotheses  1  and  2  are  satisfied.  If  pH  >  0.5, 


then  Pq|  B(a,0)  converges  to  one  ss  a  goes  to  infinity;  if  pp  =  0.5, 
then  B(a,p>  converges  to  0.5  as  a  goes  to  infinity;  and  if  p^  <  0.5 

then  Pgg  B(a,p)  converges  to  zero  ss  a  goes  to  infinity. 

Proof:  Using  the  well  known  Gaussian  approximation  to  the  binoaiial  dis¬ 
tribution,  we  have 


t2(o)  2 

lim  P„.  (u.6)  -  lim  (2n)“0,5  /  e"0,5t  dt  , 


t^a)  ■  -a(  1-q)  ( ( aq(  1-q) ) 


w  #v 


t2(o)  ■  (aCq-O^-O.SXfaqd-q))-0**  ,  a  odd, 
t2(«)  “  a(q-0.5)-l)((aqd-q))”°*5  ,  a  even  . 

It  is  clear  that 


lia  t-  (a)  -  -•  , 

<*-»• 

lint,  (a)  ■  +•  ,  q  >  0.5  , 
a->«  2 

lint. (a)  =  0  ,  q  ■  0.5  , 
a— »« 

list. (a)  ■  ,  q  <  0.5  , 

«-»•  2 


and  thus 


lia 

<2n)~°*5 

t2(o) 

/ 

c— >« 

t^a) 

lia 

(2«,-°*5 

t2(o) 

/ 

o— 

t2(a) 

lia 

(2n)~°'5 

t2(o) 

f 

a— >® 

t^a) 

-0.5t 

0 


2 

dt 


-0.5t. 
e  dt 


1  ,  q  >  0.5  , 

0.5  i  ^[  *  0.5  $ 

0  ,  q  <  0.5  . 


Lena  2  implies  that  if  a  is  odd $  P_  (a,0)  is  a  strictly  increas 
ing  function  of  o  when  >  0.5,  P_.  (a,0)  is  constant  when  *  0.5, 

UI,B 

and  Pgj  B(o,p)  is  a  strictly  decreasing  function  of  a  when  p^  <  0.5. 

Lease  3  iaplies  that  if  a  is  even,  P^  ^(a+2,0)  >  ^(a.0)  when 

p^  >  a/ ( 2 ( a+1 ) ) ,  P  (a+2,0)  -  P-.  ,,(0,0)  when  p^  -  a/(2(a+l)),  snd 


PCM ,*(°+2*0)  <  pCM,a*°*^  PP  <  «/(2(o+l)).  It  follows  that  if  o  is 
•won,  PCH,  a(a,0)  is  a  strictly  increasing  function  of  a  when  1  0.5 ; 
pai,*(o»P)  first  increases,  reaches  a  aaxiaua  value  that  depends  on  p^, 
and  then  decreases  when  pP  is  in  the  interval  (1/3, 1/2);  P_.  (4,B)  - 

Pgj  ^(2,0)  and  Ppy  a(a,p)  is  a  strictly  decreasing  function  of  a  for  a  1 
4  when  p^  »  1/3;  and  finally,  P-_,  (a,0)  is  a  strictly  decreasing  func- 

tion  of  o  when  p^  <  1/3. 

Leaaa  4  iaplies  that  if  a  is  odd,  Pgj  ^(a+1,0)  <  P^  ^(a.0)  for  all 
values  of  p^.  In  other  words,  it  is  never  advantageous  to  choose  an 
even  a. 

Under  what  conditions,  then,  can  this  error  aasking  scheae  be  used 
to  increase  our  confidence  in  the  results  of  parallel  conputations?  In 
light  of  Leaaaa  2,  4,  and  5  and  the  fact  that  P^  B(l.p)  -  Pc  m,  error 
aasking  (with  an  odd  nuaber  of  clusters)  can  be  used  to  guarantee  that 
PCj|(a.0)  >  Pg  B  if  and  only  if  p^  >  0.5,  in  which  oase  we  can  ensure 
that  Pgf(a.p)  is  as  close  to  one  as  we  wish  by  choosing  a  sufficiently 
large. 

Ve  have  just  seen  that  even  when  p^  >  0.5,  there  is  no  way  to 
ensure  that  Pg|(a,0)  “  1  using  a  finite  nuaber  of  coaputing  eleaents. 
However,  if  one  does  not  adhere  to  the  principles  discussed  in  the 
introduction,  and  is  willing  to  aake  stronger,  aore  optiaistie,  assuap- 
tioas,  then  it  is  possible  to  obtain  P(a((a,P)  -  1  with  a  finite  o.  For 
ezaaqple,  if,  in  addition  to  Hypotheses  1  and  2,  we  assuae  that  the 
amber  of  faulty  ooaputing  eleaents  is  at  aost  (,  then  the  choice  of  any 


o  .>  2g  +  1  guarantees  that  the  nuaber  of  correct  cluster  outputs  is 
strictly  greater  than  the  nnaber  of  incorrect  cluster  outputs.  In  this 
case,  the  aasker  always  chooses  a  correct  output,  and  therefore  Pgj|(a,0) 
«  1.  Alternatively,  if  in  addition  to  Hypothesis  1,  we  assuae  that  (i) 
whenever  there  is  at  aost  one  block  of  cardinality  greater  than  one,  the 
aasker  chooses  an  output  froa  that  blook,  (ii)  the  nuaber  of  faulty  coa- 
puting  eleaents  is  at  aost  g,  and  (iii)  any  two  incorrect  cluster  out¬ 
puts  aust  be  distinct,  then  any  choice  of  a  i  g  +  2  ensures  that 
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3.  EFFICIENT  COMPUTING  NETWORK  DESIGN 


Suppose  that  that  it  it  possible  to  solve  a  given  problea  in  a 
given  time  using  a  olnster  of  0  computing  eleaents  of  a  given  type,  and 
assume  that  the  basic  reliability  p  of  this  type  of  coapnting  eleaent  is 
knovn.  Furthermore,  suppose  that  ve  want  to  design  a  ooaputing  network 
consisting  of  a  such  clusters  and  an  error  masker,  so  that  the  aasker 
output  is  correct  with  probability  at  least  9  (0  <  9  <  1) .  To  aeet  this 
requireaent,  it  is  sufficient,  in  view  of  the  results  of  the  previous 
section,  to  ensure  that  the  following  inequality  is  satisfied  for  soae 
odd  integer  a  i  1: 


(a-l)/2 

Z 

J-0 


jlL 


<!■**> 


1  9. 


(9) 


Given  p,  0  and  9,  it  aay  or  aay  not  be  possible  to  satisfy  (9).  If 
(9)  oan  be  satisfied,  the  aost  efficient  design  is  obtained  when  a  is 
chosen  to  be  the  smallest  odd  integer  o»  that  satisfies  (9).  In  order 
to  analyze  the  feasibility  and  efficiency  issues,  we  partition  the  set 
of  all  pairs  (9,p^)  into  disjoint  subsets  VQ,  and  Vg  as  follows  (see 
Figure  1): 

V0  -  (<9,pP)|  pP  <  9  £  0.5  >  U  {(9,pP)l  pP  ±  0.5  <  9  ). 


▼j  -  (<9.pP)l  pP  1  9  J, 


Vg  -  (<9,p*)l  0.5  <  pP  <  9  J. 


Hm  following  less 
preceding  section 


is  i  diroot  consequence  of  the  resnlts  of  the 


LtMft.  fi:  If  (®»P^)  is  in  V^,  then  (9)  cannot  be  satisfied.  If  (0,p^)  is 
in  Vj,  then  (9)  esn  be  satisfied  and  a*  «  1 .  If  (0»p^)  is  in  Vg,  then 
(9)  enn  be  satisfied  and  a*  i  3. 

Ve  know  that  when  the  pair  (0,p^)  is  in  Vg,  the  error  aiasking 
seheae  nay  be  need  to  increase  onr  eonfidenee  in  the  ontpnt.  The  amount 
of  neeessary  replication,  given  by  the  value  of  a*,  depends  on  the  pre¬ 
cise  location  of  the  point  (0,p^)  in  the  region  Vg.  Figure  2  shows  the 
decomposition  of  Vg  into  snbregioas  for  which  a«  equals  3,  3,  7,..., 
etc.  These  regions  are  relatively  narrow,  and  become  smaller  as  a, 
increases.  It  follows  that  small  changes  in  p^  nay  lead  to  large 
changes  in  a*.  Ve  now  present  an  example  that  illustrates  this  situa¬ 
tion. 

&UMUlS.:  Suppose  that  we  want  to  ensure  that  the  probability  that  the 
output  of  the  masking  scheme  is  correct  is  at  least  0.93.  This  con¬ 
straint  will  automatically  be  met  if  (9)  is  satisfied  with  9  “  0.93. 
Thus: 

if  0.930  £  p^»  then  o«  *  1,  and  no  replication  and  error  masking  is 
needed, 

if  0.833  i  pP  <  0.930,  we  need  a,  ■  3, 

if  0.811  i  p^  <  0.833,  we  need  a,  ■  3, 

if  0.773  i  p^  <  0.811,  we  need  a,  ■  7, 

if  0.749  i  p^  <  0.773,  we  need  a,  -  9, 


v  v 


A'  -.'V-' 


Vl%‘  - 


Vw'v  ■  • 


if  0.729  i  F  <  0.749.  we  seed  a,  -  11. 

•ad  so  forth.  It  is  clesr  that  the  necessary  saoaat  of  rspliestioa 
iaoressss  quits  rapidly  with  decreasing  pP.  Suppose  aov  that  vs  i 
priori  fix  a  to  be  equal  to  3  (the  triple  aodulsr  redundancy  case). 

Then,  in  order  to  guarantee  that  PggO.f))  >  0.95.  ve  nuat  have  pP  1 
0.865.  For  a  given  p  (that  is.  for  a  given  type  of  ooaputing  elenent). 
this  translates  into  an  upper  bound  on  the  number  0  of  ooaputing  ele¬ 
ments  that  oan  be  used  in  eaeh  cluster;  in  particular. 

P  *  P*  log  p 

where  x  is  defined  as  the  largest  integer  less  than  or  equal  to  x. 

Thus,  if  p  -  0.99.  0*  -  14;  if  p  -  0.95,  0*  *  2;  and  if  p  *  0.90,  0*  - 
1.  A  limit  on  0  is  clearly  a  limit  on  the  cos^utational  speed  of  the 
network,  thus  emphasizing  the  fact  that  the  speed  and  reliability  of 
parallel  processors  are  interdependent.  Finally,  it  is  interesting  to 
ask  what  minimum  computing  element  reliability  p  is  required  in  order  to 
use  a  large  nusd>er  of  elements  in  eaeh  cluster,  while  ensuring  that 
*01(3.0)  >  0.95.  Since  we  need  pP  1  0.865. 

P  1  Pe  -  (0.865)1/P. 

Thus,  if  0  -  256,  p,  -  0.9994;  if  0  -  1024,  p*  -  0.9998;  and  if  0  - 
2048,  p,  -  0.9999. 


4.  CONCLUSION 


The  results  of  this  paper  show  that,  when  reasonable  assumptions 
are  made,  striet  hardware  redundancy  is  of  limited  value  in  meeting 
parallel  processing  reliability  constraints.  If  the  reliability  of  the 
basic  computing  element  is  not  high  enough,  it  is  impossible  to  achieve 
the  desired  network  reliability  using  strict  redundancy  (Figure  1, 

Region  V^) .  If  the  reliability  of  the  basic  computing  element  is  suffi¬ 
ciently  high,  no  redundancy  is  needed  to  achieve  the  desired  network 
reliability  (Figure  1,  Region  V^),  The  remaining  case  (Figure  1,  Region 
Tjj)  is  the  only  one  in  which  strict  redundancy  is  of  use.  However,  the 
amount  of  replication  needed  to  satisfy  the  reliability  constraint  may 
be  quite  large,  as  illustrated  in  Figure  2. 

In  view  of  the  preceding  discussion,  it  is  clear  that  an  alterna¬ 
tive  approach  to  meeting  reliability  constraints,  without  massive 
amounts  of  hardware,  is  desirable.  The  authors  are  presently  working  on 
such  an  approach  in  the  context  of  signal  processing.  The  basic  idea 
consists  in  taking  into  account  the  structure  of  the  problem  to  be 
solved  and  in  exploiting  any  inherent  redundancy  in  the  input  data 
[MBT82] . 
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